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A telecommunications system with multiple modes of inter- 
facing with users. The device accepts, for example, speech or key 
input and outputs both graphical display data and vocal data. A dis- 
play at the user site displays various communicadon options to the 
user such as to call a number, cull by name, or look at a directory 
of names. The user site also includes a voice processor fli^ speala 
infoimation reflecting the status of the telecommunicadon system or 
reflecting the Infonnadon on the display. 
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MULTIMODAL USER INTERFACE WITH SPEECH IN-/OUTPUT AND GRAPHIC DISPLAY 
pFT.ATRD APPTJCATIONS 
This application is related to U.S. patent appUcation, Serial No.08/841,485. 
entitled ELECTR014IC BUSINESS CARDS; U.S. patent appUcation, Serial No. 
08/842,015, entitled MULTITASKING GRAPHICAL USER INTERFACE; Serial 
. ..No..08/08/841,486, entitled SCROLLING WITH AUTOMATIC COMPRESSION 
AND EXPANSION; U.S. patent application. Serial No.08/842,01 9, entitled 
CALLING LINE IDENTIFICATION WITH LOCATION ICON; U.S. patent 
application. Serial Mo.08/842,01 7. entitled CALLING LINE mENHFICATlON 
WITH DRAG AND DROP CAPABILITy; U.S. patent appUcation, Serial No. . 
08/842,020, entitled INTEGRATED MESSAGE CENTER; and U.S. patent 
application, Serial No.08/842,036, entitiedlONIZEDNAMELIST, all of which were 
filed concuirentiy herewitii. and all of which are hereby incorporated by reference. 

This invention relates generally to die field of tdecommunications sqmpment, 
and more specifically to tiie speech and graphical user interfaces for 
telecommnnications equipment tiiat facititates the entry of input commands. 

' Telecommunication systems are available witii a speech-recognition capability 
for performing basic tasks such as directory dialing. Additionally, tiiere are network- 
based speech recognition servers that deliver speech-enabled directory dialing to any 
telephone. Botii of tiiese types of applications use discrete or non-integiated 
techniques. That is, tiiey use eitiier a graphical interface or a speech interface but not 

■ ■■■■iMtt- .. 

While speech interfaces have been around for a number of years, tiiey have not 
gained widespread acceptance. Speech interfaces are difficult to use for several 

reasons. One reason is lhatthe new user has no idea what is acceptable grammar or 
input vocabulary at any given time in a dialogue. For instance, tiie user may say. 
: >'Thohe John'\ whereas tiie recognizer ma^ only accept "Call John", or "Dial John" 
■ ; : . • Also, tiie user often does not know when the recogn^ Users 
/rnay talkwhentherecogmzErisoff, and^to 
... response, / ■ 
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b addition, the best available speech recognizer, have recognition 
per£brmancebetwcen90and95percentnnderideaico„diticn3. Generally condittons 
arenotidealandperfonnanc=™llbeafl:ectedby,forexample.anoisy enviiomnenV 
otespeaker.,n.eraccents.oran^speakingtoosoffly,Witbaspeechinterface, 
tese poor conditions can be handled though additional dialog. The speech 
recognizermaygivetheuseradditionalinstmcUonsanda^^^usertorepeatlhe 
utterance. Using apeechto provide addidonalinfonnation,od>eu«r is very dow. 
especiaUywhenmulflpleoptionsareinvolved, This can result in a tedioua and 

frustrating interaction. 

Generally, speech is fast for input and slow for output, m addition people 

forget^hatwassaid, First, if speech is used to present the user »id. a Us. of cho.ces. 
theywiUlMyhaveforgotten^eflrstchoicebeforetbeendoftelistisreached. 

lhisisaconunnnproblem.wthint=ractive.voice-respons^ 
: second.ifsp=echisuscdtogivedetailedinstrucfions.«»,nsermustrelyonmemory 

to recallany of theinfonnation. Third, users often become Mosf if. speech 
„Ucatioosbecausetheydonottaowwhallevel.theyareat,or«brtmenui*:msare 

. available. • s 

Therefere,an=ed exists for amultimodal interface including a combmahon of 
speech and graphical inter&cesaUowgausertcefBcienUy initiate and cornplete 
tasks. Theusern^^tbeabletoeasUychoosetomosteffidentrneansofinteractmg 
/ with the telecommunication system. 

Systemsandmethodsconsistentwifhthepresentinventionaddressthisneed 
hyprovidiriBarndtimodaluserinterfacethatprovidesauserwiftmorethanone 

input device for efficient entry of connnands to a system. 

Inaccordance^iththepurposeoftheinventionasembodiedandbroadly 

: describedherein, the multimodd user interfaeeconsistent^iftep^^^^ 
. presentinvcntionrndudesatelecommunicationssystemvaUtmultiplemodesof 
interlaciug ,»ftuse«.including-voice. hard key.touch input, peninput. etc. The 

' : device accepts vocal or key input and butputs both graphical display dataand vocal 
W Adisplayatteusersitedisplaysvariouscon>mnnica.ionoptionsto.heuser 
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Huoh » caU a number, cdl by narne, or look a. a m of names. Tb= user site also 
Mudesavoiceprocessorfhat speaks mfonn^iionreflectiBg the s.am.of*e system 

or reflecting the infcnnafioil on to display , 

EITTF nrPfTTTP"'" AWMGS 
The accompanying drawings. whichareincorporated inan^ 
. ofthisspedfication,Ulus»te.ys.emsBndmetodsconsistentwiftteinv=nnonand. 
together wi4 the description, e^lain the objects, advantages and principles of d>= 

invention. In the drawings. 

Figure 1 is ablock diagram of a commvmications network operating m 

cor^^ctionvriti^ther^titaskinggrapMc^^ 

invention; ■, r 

FignreliaadiagramofansermobiletelephoneoperatinEhithenetworkof 

Figure 1; wi 
FigureSisablockdiagramoftiieeb^^ 

telephone of Figure 2; ■ ' t, a i 

: Figure4isabiockaagramof1hesoftwar^comp^^^ 

ROM of Figure 3; 

Figure 5 is aUock diagi^ of the graphical user iirtc^ 

Figure 4; . 
: : Figures 6-9ai.flowchartsshowings^^ 

■ requests according to the present invention; : : 

; : : Figures lOa-lOfareexamplcscreendisplaysaccordingtoth^ 

invention; and 

- Rgi^ll is ^example directory accordm^ 

The foEo^ng detailed description of to invention refers to to accompanymg 
ta™g.totfflustratepreferredembodimen,sconsistent*ithtoprineiplesoftas 

: : kventior. Otorembodimentsarepossible^^ , 

: : Mowingdetailedde^pdond^^ bs,ead.toscopeofto , 

; inyendon is defined only by to ; , ■ : 
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Th=— ial.ysten,otthepresentto,ntionc=nbe..ed,ooveroo,nea 

,„.„ac.o. T..speecK»odality.a„be^df..fastha:^-^=»d^«^^^ 

3uoha.caUingap=:son.h..drivi„Sacax. '—^^^^ ^^f^^ , 

:i„t«face.g.=pMcalfeadbadccoddb.Wtopre,.n.— .cho.«s«^ 
,.,..;L^..as.ow^ehna.e*.^==h..o^e.^*=-.a..) 
i^a.i^a..rtto,^m.u,=rV.ow.h=nt„tal.a„dwh.«.Hs.n,ctt^^ 
.cogniz..di.p.ayt=.ttob.fteu=erl=ooware*ea.c=p«d™«=nd 
oomn.a„d«ord.,andtodisplayt«tandgraphics«™n«vu^rstaougha 

multimedia tutorial. 

Fig^elisablockdiagrBmofa=o.™unioationsn=^orkc.n.aiBmEmobUe 
»=n00baving*=m.Mta3Hn.^aphi«l-ain.=tfacc^c^^^^^^^ 

i™,udi„gext.n^s=rve.aBdda.ab.=s,.chasnetwo*aervic.pn>v,^ 

using mobUe telephone 1100. „ ^,„,n^„ 

Th=«s.aho»=esmobae.d=phoneUOO.cco— tcw^thcaU^ia™^ 

" M..n,typ=sofcon>mmacaaooe,mp»en,sucha.ordi.aryt=l=phcna300^ 
: : .„Me..ephoneUOO.«biobUsi^ar.o«s=rn,.bne^^^^^^ 

=,^po.=nt 1500, computer .600, and Analog Display Sovicos latcrftce (ADSI) 

: cal.eroonnnudcaUone,uipo.n.l300taoughl700overaco— 

: : : QSMp.o.colwMch.d.orib=dm4er.a^appUc=tioo.lis.=^ 

the application. ' , . nnn 
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1700 normdly comect to GSM switching fabric 1800 Via another t^^ 
such as a PubUc Switched Telephone Network (PSTN). 

The user communicates with a caller or network services provider 1200 by 
establishingeitheravoicecalloradatacallGSMnetworksprovideanerror-ftee, 

guaxar^eeddeBverytransportmechamsmbywHchcaUerscan^^^ 

point messages. . 
MobUet=lephonell00provides.user.6iendlym,erfaceto&cffim«,nconung 

„do».gomgconuB™cadonbyfhe,.er. Figure 2 is . di^gtan, of mobile .eleph.ne 
„OOftatope«.=sin*eaetwoAsho™mFigur=l.M.bil=.=.ephoBellOOindud. 
m»iB housing 2100, keypad2300, display 2400,^d toem^gportion 2500. 

Figure3isablockdi.giamoffhetard«BreelementsinmobUetdephone 
1100 i.cludi.gaateuua31C0.co™nunic.Uousmodule3200,featuieproc=ssot3300, 
memory 3400, sliding keypad 3500. andoE conWUer 3600. dispky m 

baW pa:!' 3800, aal switching power supply 3900. 

Alltel 3100 tiansinits and receives, radio ftequer-cy in^^ 
tdephonellOO. Ante-ma 3100 preferably comprises aptainvertedF antenna 
(PIFAj-typeorashortstub (2to4cm)=ustomhelixantenna. Antemm3100 
conununicates ov« GSM switching fabric 1800 using a conventional voiee B- . 
d^el, dataB-chamel, or OSM signaling channel connection. 

Commnnicatipnsmodule3200»^^^ 
GSM radio, basebaiii and audio fimctionality fcr mobiletelephone llOO. 
Co.nmuuicaticnsmodnle3200includesGSMradio3210;vEOA3230.BOCK3250. 

and audio transducers 3270. 

GSMradio 3210 converts ti>e radio ftequency information toft^^ 
i,toanalogbasebandinfcrmationforpresentationtoVEGA3230.VEGA3230is 
preferablyaTexasInstrumentsVEGAdevice,containinganalog-to-digital 
(Affl)Migital-to.analog(D/A)conv=rsion™its3235. VEGA 3230 converts 

^ . 

presentation to BOCK 3230. ^ 
: rccnventionalAEMrMcroprocessora^ 
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- 3250 performs GSM baseband processing for geueratmgdigM 

supporting GSM protocols. BOCK 3250 supplies the digital audio signals to VEGA 
3230 for digital-to-analog conversion. VEGA 3230 applies the analog audio signals 
to audio transducers 3270. Audio transducers 3270 include speaker 3272 and 
microphone 3274 to facilitate audio communication by the user. 

Feature processor 3300 provides graphical user interface features, voice user 
interface features, and a Java Virtual Machine (JVM). Feature processor 3300 
communicateswithBOCK3250usinghighlevelmessagingoveranasynchronous 

(UART) data linlc. Feature processor 3300 contains additional system circmtry, such 
as a liquid crystal display (LCD) controller,. timers, UART andbus interfaces, andrcal 
time clock and system clock generators (not shown). 

Memory 3400 stores data and program code used by feature processor 3300. 

Memory 3400 mcludes static RAM 3420 andflashROM 3440. Static RAM 3420 is a 
volatilememorylhatstoresdataandotherinformationusedbyfeatureprocessor 

3300. Flash ROM 3440, on the olher hand is a non-voladle memory &at stores ^ 

program code and directories utiUzed by feature processor 3300. 

Sliding keypad 3500 enables the user to dial a telephone number, access 

remote databases and servers, and manipulate die graphical user interface features. 
Sliding keypad 3500 preferably includes a mylar resistive key matrix that generates 
analogresistivevoltageinresponsetoactionsbylheuser. Slidmg keypad 3500 
preferably connects to mainhousing 2100 CFigore 2) of mobile telephone 1 100 

tlnough two mechanical "push pin"-type contacts. 

Analog controller 3600 is preferablyaPhilhpsUCBllOO device that acts as an 

interface betweenfeature processor 3300 and slidmg keypad3500. Analog controUer 
3600 converts the analog resistive voltage from sliding keypad 3500 to digital signals 
for presentation to feature processor 3300. 

Voice processor 3550 receives voice commands from a user speaking into 
microphone 3274. Itattempts to depode ihe command using ^^^^ 

systems and metliods. 

Display module 3700 is preferably a 160 by 320 pixel LCD vdth an analog 
touch screen overky and a» =lecw>l—enl bacMght Pis^^^ 
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operatesmconjnnctionwithfeature.processor3300todi^^^^ 
interface features. 

Battery pack 3800 is preferably a single UtMum-ion batteiy with active 
. protection circuitry. Svvitching power supply 3900 ensures highly efficient use of t^^ 

■ liihium-ion battery power by converting the voltage of the lithium-ion battery into 
stable voltages used by the other hardware elements of mobUe teleplione 1 100. 

Figure 4 is a block diagram of the software components of flash ROM 3440, 
including interface manager 4100, user applications 4200, service classes 4300, Java 
environment 4400, real, time operating system (RTOS) utilities 4500, and device . : 
drivers4600. 

: Merface manager 4100 acts as an application and window ma^^^^ Interface 
manager 4100 oversees the user interface by allowing the user to select, run, and 

. otherwise manage applications. 

User appUcations 4200 contain all the user-visible applications and network 

, service appUcadoD.. User applications 4200 preibrably Inchide ^ 

application for processing incoming and outgoing voice calls, a message processing 
application for sending and receiving short messages, a directory management 
application fbrmanagmg database entries in^t^^ 

application, and other applications. 

Service classes 4300 provide a generic set of application progranommg 

faciUties shared by user appUcations 4200... Service classes 4300 preferably include 
. varionsntilitiesandcomponents,suchasaJavatdephonyappUcationinterface.a 

voice and data manager, directory services, voice mail components, text/mk note 
^ ■ : ;components, e-mail components, fax components, netwo± services management, and 

other miscellaneous components and utilities, . 

Java environment 4400 preferably includes a JVM and the necessary rrni-time 

. - hbraries for executing a^^^^ 

; ■ ' : in^S utiUties 4500 provide red time tasks, low le^^^ 
V ■ in^lementations to^support Javaenviromnent 4400, RTOS utUities 4500 preferably ; 
: inchadeJavapeers;suchasnetwor^ . 
; engines requiring detailed real time control and high performance, such as recognition 
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engmes and speech processing, and standardumiti^^^ 

managers, and database packages. . 

- Device drivers 4600 provide access to the hardware elements of mobile 
telephone l lOO. Device drivers 4600 include, for example, drivers for sUding keypad 
3500 and display module 3700. 

: Feature processor 3300 executes the program code of flash ROM 3440 to 
providetheuserfriendlyinterface. Merfecemanager 4100 controls the graphical user 
interface and the voice interface. In one embodiment of the present invention, the 
speech recognition software appUcation is BM' s Voice Type AppUcation for 
Windows running on a standard Pentium desktop computer. However, other voice 
processorsmaybeused. The speech recognition software can be eiiJier in the device 
itself or onanetwork-based server remotely accessed bylhe device. 
: . . Figure5isablockdiagramofinterfacemanager4100,m^^ 
• «er 5100, configuiBtionmanager 5200, and applications manager 5300. The 
interface manager uses standard programming 1^^^^ . 
languages. 

System manager 5100 acts as atop level manager. Configuralion manager 
5900handlesthedatamanagementforthesystem. Applications manager 5300 
/ manages user applications 4200. AppHcations manager 5300 handles the starting and 
^stopping of user visible appUcations, display access, and window management. 
Applications manager 5300 provides a common ^pUcation framework, appHcatipn. 
and applet security, and class management 
. V::system manager 5100,^^^ 
^ ^ :. 5300 worktogether wi4^^ of interface manager 4100 to provide the 

: ■ enviromnentto allow theuser to sdect,rmi, an^ 

- : eitheragraphicalinterfacecravoiceinterface. Interface manager 4100 provides a 
. processor-3550 andtlie speaker/receiver on the telephone 2100. : 
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n. ^ygtffm Processing. 

Figures 6-9 are flow charts showing steps the interface manager 4100 may 
performto cany outmethodsconsistentwith the present invention. Figures lOa-lOf 
.how example screen displays according to one example of the present invention. 
Figure 1 1 shows a directory with called party data. 

Systems and methods consistent with tlie present invention provide both a 
■graphical and voice interface for use to initiate and process telecommunications. A 
. caller may enter commands and data either vocally or using a keypad or some other 
manual input device. The caller will receive feedback ftom the telecommunicati^^ 
system both vocally.and graphically.. This allows the user to choose the most 
convenient method of interfacing with the telecommmiications device. 

. AnembodimentofthepresentinventionwiUnowbedescribedwithrespectto 
Figures 6-11. The steps in the flow charts include example information for displ^^ 
display screen2400andfbrvocaUzationover speaker 3272. All references to display 
refer to display on screen 2400. all references to voice input refers to^ 
3274andvoiceprocessor3550,andallreferencestospokenou1putrefertospeaker 
3272. Display information is represented with a "G" for graphical and sound 
informationis represented with "S" for sound. Commands, represented by "C", may 

be input by the user usmg any known input device. 

The specifics of what is spoken by the system or what is displayed are merely 

exemplary, One of ordmary skill inthe art would recognize that many difierent 
display information or spoken information may be mcluded. In addition, the graphics 
. andorvoicemaybetumedoffattheuser'sconvenience. The order of the steps may 
. be altered without affecting the basic system, wMch ^^^^ 

giaphical and vocal output and mputto aUo^^ 

To initiate commmncations processing consistent withthe.Fesent 

V an attention word suchas'-start'Hs pref^^^ 

begm. As shown in Figure 6 the phone system 1100 awaits the attention word or key 
input before initialing some telecommmncation action (step 600). The user may input 
. an attention word or command using any known input device such as verbaUym^^ ■ 
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microphone 3274 for processing by voice processor 3550, manually using the keypad 
3500 or pressing on a touch sensitive screen. 

When the user. speaks a word or presses a key (step 605), the system must first 
recognize the key or the key word as being an attention word/key (step 61 0). If it is 
not, the system remains in the state of waiting for the attention word or key mput 
(step 600). Once the key is recognized, the system acknowledges receipt of the key 
word or key input by an audible sound and the graphical display 2400 will display, 
' and the sound portion 2500 wUl speak, various choices for the user such as call name, 
call number, directory (step 61 5). The directory option refers to reviewing or 
maintainmg a directory ofpotential called parties, such as is currently known in the 
art. The system enters a wait state waiting for a command (step 620). 

When a command input by the user is not recognizable (step 625), the system 

■ notifies the user of this lack of recognition. For example, the system may say 

. , "pardon" to the user and display the request to either call name, call number, directory 

■ ' (step 630). . 

The user may enter a command to call a specific number (step 645), thereby 
initiatmg the call number fimction steps shown in Figure 7 (step 700). If the user 
. enters a command to call a specific named person (step 640) then Uie call name 
fimction steps shown in Figure 8 are performed (step 800). When the user enters a 
command to access a directory (step 635), then the system wiU perform known 
, .. directory fimctions (step 1100). 

Typically, the wait state of step 620 will last a predetermined amount of time, 
such as three seconds, and if no input is received (step 650), the system will display 
,: Wasktheuserverballytomputwhattypeof command theywishto entersuchasa 

command to call a specific name, phone number or to review a directory of names 
' (step 655). Processmg then returns to the command wait step 620. However, if no 
command is input by the user again within the predetermined amount of time (step 
650), the system will go back to step 600 and await another attention word or key. 
. . . Figure 7 shows the steps performed by the call number fimction 700. First, the 
V There may be several 

: different numbers of digits that are acceptable. For example, for calling an internal 
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number, three digits may be acceptable. For calling a local number, seven digits insy 
be acceptable, and for calling a long distance number, eleven digits may be 
acceptable. If an incoirect number of digits is entered, the system will verbally state 
to the user "pardon" and display an error message requesting that the user input a new 
number (step 710). processing continues with step 705, 

: If an acceptable number of digits is entered, the number is called. The system 
. wUl audibly state to the user that the number entered is being caUed, and the display 
will show the number (step 725). Before calUng, the system pauses and listens for an 
indication from the user that he does not wish for the call to proceed (step 730). If the 
user never requests the change (step 735), the user will hear the DTMF sound of the 
numbers being dialed, and the system will display during the phone call the choices of 
selecting to hold or hang up (step 736). The conversation proceeds (step 737) until 
the user either selects to hold or hang up (step 738). . : 

Returning to step 730, the user may take some action to interrupt the initiation 
. of the phone call. If the user says a word that is not recognized (step 740), the system 
■ prompts the user to say whedicr they wish to call the currently displayed party or 
. number (step 780). If the user says yes, then the procedure of calling the displayed 
party or number continues (step 785). Otherwise, the system will again state and 
display the users basic options of call name, call number, or directory (step 790). 

If, during the waiting period step 73 0, the user inputs a new command such as 
call number, then the call number routine is begun (step 800). If the user mputs a new 
command to call number, the system restarts processing with step 705. Finally, if the 
user just gives an indication that this is not the correct number (step 745), the system 
prompts the user to input a name or number to call (step 760). If the user wishes to 
. call a number (step 765), processing restarts with step 705. If the user wishes to call a 
name (step 770), processmg continues with the call name routine (step 800). 

The call name function 800 will be described with respect to Figure 8. First, 
the system evaltiates thename entered by the user (step 805). To evaluate the name, 
the system will look to a directory that includes a list of names and numbers and other 
identifying information. The directory may be stored in meniory 3400 or may be on a 
server on the network. An example directory with directory entries is shown in Figure 
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1 1 . As shown, many pieces of information about a party may be stored including the 
name, title, organization and address. Phone numbers are provided each of the 
different locations or types of communication devices associated with the party shown 
in the icons column. Tlus allows auser to direct not only the name of the person to 
call, but also to where they should be contacted or on which communications device 
they should be contacted. The directory may be reviewed and edited using known 
data processing systems- 

If a name is not in the directory (step 810) then the system will verbally ask 

the user to repeat themselves, such as by stadng "pardon," and will graphically request 
the same information (step 811). The system wiU then wait for the next user 
command (step 812). If, after a given number of times, such as three times, the name 
provided by the user is still not recognized, tiaen the system will verbally request the 
user to give a different name or to add this person to their dkeclory so that they may 
call the person (step 814). If the user selects to add the name to a dkectory then the 
add name data processing procedure known in the art will be performed (step 815). If 
the user still says nothing or says the wrong piame. the sysiem will retiim to. its initial 
state of Hstemng for the attention word 600. If the met enters a new command, it is 
performed (step 816). 

Returning to evaluating step 805, if the user enters multiple names or locations 
(step 900), the processing will continue with the procedure shown in Figure 9. If the 
name is evaluated and recognized, the system will state that it is calling the named 

. person and the graphics will display the same (step 820). When a location is specified 
along with the called party's name, the system will state that it is callmg the named 
person at a given location and the graphics will display the same (step 825). The user 
then has a chance to change his or her mind and may enter a change to the displayed 
called party (step 730). Processing continues as shown in Figure 7, allowing the user 
a chance to change the currently displayed called party or to continue processing. 

■ Figure 9 shows the steps of the fionction called when a user enters a name that 
sounds like many others in the directory or when the user enters a name that has a 
plurality of locations associated with it in the directory, The system determines 

. whether there are multiple names that match or might match that input by the user 
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(step 910). If ». fte sv.t=m ^.ks user which of peopk to caU, to system 
wUldisplaytolistofi™.=s(step915). Ifthemer enter, the command to call a 
speciSc name (step 920), the syatem wiU oonto processing by gomg B step 820 

(step925). . '. ■ 

,f,here„=nctmultipl=names(step910),tother=areaadtiplelocada„sm 

the directory for the names party . Wore, the system displays a list stored m the 

^ectoryfiomwhichfheusermayselectalocationtocalltheparty(step930). The 

system wiU then audibly state that it is calling a specific name at a specific locatton. 
andftesameisdispla,edCstcp945).Proces,ingcontinnes™thstep730assbownm 

Figure 7- 

Figures lOa-lOf show example screen displays accordingto fhepresent 
invcndon. Figure 10a shows te basic screen display «flr1he users selections to dial 
byname 100crbyntMrber200. n.e name Ust selection 3O0 allows fte user ,c v,ew 
thedircctoryofnames.suchasti,edirectotyshowninFig«reU. After an a«enbon 
wordisenieredinto*esys.em.icon300showninFis..-=10bisdisplayedonfl.e 
screen to indicate to the user that the system is on and waiting for acommand.. 
Tl„oughou.processingthetelephonecall,icon300isdisplayedwheneverittsbme 

for user input. , 

Icon 400 shown in Figure 10c indicates to ttte user ftat the system tsprcv.dmg 
display and vocalontput.lnthis sample screendisplay^enserinpntfte command 

to caU grandma and the system is displaying the two enMes 402, 404 in the directory 
tomatchflae revest. FigurelOd shows thenser touching the tonch sensitive screen. 

500 «. select one grandma. Fi^ lOe shows an example display showing the name 
and number oftecurrently being calledparty.FignrelOfshows.he screen displayed 

tofteu^rafterconnectionwiththecaUedpar^. As shown, tenser may ri==. to 

: place the calledpartyonhold or hangup.^ ^ ■ - ■ : 

nl. <7 n"clusion. 

The combined speech and graphical user mterfacc consistent .. 
principlesofthepresentinvcntionprovidesasimplcinteracUonm^ 
^er can sdect and operate conmimication tasks with ease. 



The foregoing description provides illustration and descriptiDn, but is not 
intended to be exhaustive or to limit the invention to tlie precise form ^^^^^ 
Modifications and variations are possible in light of the above teachings or may be 
acquired from practice of the invention . 

Additionally, the foregoing description detailed specific graphical user 
interface displays, containing various graphical icons and buttons. These displays 
have been provided as examples only. The foregoing description encompasses 
obvious modifications to the described graphical user interface displays. The scope of 
the mvention is defined by the claims and their equivalents. 
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WMATTSSnT.ATMEDIS: :. 

1. A communication unit comprising: 

means for displaying communication information prompting a caller for input; 

means for speaking audio communications reflecting the displayed 
information; and 

means for receiving vocal or manual data input &om a caller providing a 
communication request. 

2. The unit according to claim 1, wherein the means for displaying includes: 
means for showing a plurality of communication options on a visual display; 

and 

wherein the means for speaking includes 

means for vocally identifying the plurality of options. 

3. The unit according to claim 2 fiirther including 

: means for receiving a selection of one of the. displayed options; and 
. means for vocally repeating tlie plurality of selections when no selection is 
received within a predetermined amount of time. 

4. The unit according to claun 2 wherein Hie means for receiving vocal or manual 

data includes 

. means for recognizing a vocal command; and 

means for requesting the caller to repeat the vocal command, when the 
recognizing means does not recognize the vocal command. 

5. The unit according to claim 4, further including 

means for maintaining a directory ofpotential called parties, the directory 

maintaming both a vocal version of the name, the text of the name, and the telephone 
number associated with the name. 

6. The unit according to claim 5 further uicluding 

means for addm^ 

7. The unit according to claim 6 fijTther including 

: " means for receiving a command to call a party with a specific name; 
means for searching the directory for the specific name and 
. calling a number associated with the specific name in the d^ 
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8. The unit according to claim 7 fiJrther including 

means for maintaining in the directory a plurality of telephone numbers 
associated with a smgle name, each of the telephone numbers corresponding to a 
different identified location; and 

means for receiving a name and location of a called party. 

9. The unit according to claim 2 further including 
means for receiving a name of a party to call; and 

means for dialing a number associated with the received name. 

10. The unit according to claim 9 fiuther including: 

means for displaying aname of a called party currently being dialed; 
means for receiving an mdication to end the current call; and . 
means for disconnecting the telephone in response to receiving the indication. 

1 1 . The unit according to claim 2 forther includmg 
means for receiving a number to call; and 

■ . means for diaimg the number. 

■ 12; The unit according to claim 11 f^ 

. means for displaying a number currently bemg dialed; 

. . m^ 

means for discomiectmg the telephone in response to receiving the indication. 

13. A method of interfacing with a communication unit comprising the steps of 
. displaying commumcation infomiation prompting a caUer for input 

speakmg audio communications reflecting the displayed mformation; and 
receiving vocal or manual data mput from a caller providing a communication 
request. 

14. The method according to claun 13, wherein the step of displaymg includes the 
step of showing a pluiahty of commumcation options on a visual display; and wherein 

■ the step of speaking includes the step of vocally identifying the plurality of options. 

■ 15. The method according to claim 14 ftirtherinchiding the steps of 

::- - receiAdng a selection of one of the di^^ ' 

vocally repeating the phiraHty of selections when no selection is rec^^^ 

, within a predetermined amount of time. 
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1 6. The method according to claim ! 4 wherein the step of receiving vocal or . 
. manual data includes the steps of 

recognizing a vocal command; and 

requesting the caller to repeat the vocal command when the . 

command is not recognized. 

17. The method according to claim 16, further including the step of 

■ maintaining a directory of potential called parties, wherein the directory 
maintains both a vocal version of the name, the text of the name, and the telephone 
number associated with the name. 

18. The method according to claim 1 7 further including the steps of 
receiving a command to call a party with a specific name; 

searching the dkectory for the specific name and calling a number associated 

with the specific name in the directory, 

19. The method according to claim 1 8 fiirther including the step of maintaming in 
the directory a plurality of telephone numbers associated with a single name, wherein 
each of the telephone numbers corresponds to a different identified location; and _ 

receiving a name and location of a called party. 

20. The method accordmg to claim 14 further includmg the steps of 
receiving a lame of a party to call; and 

dialing a number associated with the received name. 

21. The method according to clami 20 further including the steps of 
displaymg a name of a called party currently being dialed; 
receiving an indication to end the current call; and 
disconnecting the telephone in response to receiving the indication. 

22. The method according to claim 14 further including the steps of 
receiving a number to call; and 

dialing the number. 

23. The method according to claim 22 fiJTther including 
displaying a number currently being dialed; ■ 
receiving an indication to end the cunent call; and 

.. disconnecting the telephone in response to receiving the indication. 
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24. A communication network comprising: 
user communication site including 

means for displaying communication information prompting a caller 

for input; 

means for speaking audio communications reflecting the displayed 
information; and 

means for receiving vocal or manual data input from a caller providing 
a communication request; and 

■ network communication site including 

means for performing the coiimiunication request. 

25. The network according to claim 24, wherein the means for displaying 
includes: 

means for showing a plurality of communication options on a visual display; 

and 

wherein the means for speaking includes 

means for vocally identifying the plurality of options. 

26. The neuwork according to claim 25 wherein the network site further includes 
means for receiving a selection of one of the displayed options; and 
means for performing the selected option. 

27. The network according to claim 24, said user site fTirther including 
means for maintaining a directory of potential called parties, the directory 

maintaining both a vocal version of the name, the text of the name, and the telephone 
number associated with the name. 

28. ThenetworkaccordingtocIaim24,saidnetworksitefiirtherincluding 
means for maintaining a directory of potential called parties, the directory 

maintaining both a vocal version of the name, the text of the name, and the telephone 
number associated with the name. 

29. The network according to claim 28 fiirther including 

means for receiving a command to call a party with a specific name; 
. means for searching the directory for the specific name and . . . 
caUing a number associated with the specific name in the directory. 
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30. The network according to claim 28 further including 

' means for maiutairiing in ±e directory a plurality of te^ 
associatedv«thasinglename,eachofthetekphonenumbersco^^^^^^ 

different identified location; and 

means for receiving a name and location of a called party. 
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